Modelling the dynamics of multi-agent learning has long been an important research topic, but all of the previous works focus on 2-agent settings and mostly use evolutionary game theoretic approaches. In this paper, we study an n-agent setting with n tends to infinity, such that agents learn their policies concurrently over repeated symmetric bimatrix games with some other agents. Using mean field theory, we approximate the effects of other agents on a single agent by an averaged effect. A Fokker-Planck equation that describes the evolution of the probability distribution of Q-values in the agent population is derived. To the best of our knowledge, this is the first time to show the Q-learning dynamics under an n-agent setting can be described by a system of only three equations.

Let me start with a global comment. I enjoyed very much reading this paper. I found it well written (apart from typos, and some English sentences constructions that are a bit heavy) and interesting. It is related to a modern sub-field of reinforcement learning: multi-agent learning, that lacks theory w.r.t. to single-agent RL. The paper introduces a mean-field analysis of a large population of agents playing simple symmetric matrix games against each others, so that, as the population gets large, each player effectively plays against a single "mean" player.

This paper introduces a mean-field model of multiagent Q-learning in repeated symmetric games. The model assumes that at each time step each agent plays symmetric games with m other randomly chosen agents, and considers the limit of n, m to infinity. Under these settings the authors have derived the Fokker-Planck equation governing the time evolution of the distribution of the agents' Q-values. The review scores exhibited quite a large split. Two reviewers rated this paper well above the threshold, whereas Reviewer #1 rated it negatively.

Modelling the dynamics of multi-agent learning has long been an important research topic, but all of the previous works focus on 2-agent settings and mostly use evolutionary game theoretic approaches. In this paper, we study an n-agent setting with n tends to infinity, such that agents learn their policies concurrently over repeated symmetric bimatrix games with some other agents. Using mean field theory, we approximate the effects of other agents on a single agent by an averaged effect. A Fokker-Planck equation that describes the evolution of the probability distribution of Q-values in the agent population is derived. To the best of our knowledge, this is the first time to show the Q-learning dynamics under an n-agent setting can be described by a system of only three equations.

Modelling the dynamics of multi-agent learning has long been an important research topic, but all of the previous works focus on 2-agent settings and mostly use evolutionary game theoretic approaches. In this paper, we study an n-agent setting with n tends to infinity, such that agents learn their policies concurrently over repeated symmetric bimatrix games with some other agents. Using mean field theory, we approximate the effects of other agents on a single agent by an averaged effect. A Fokker-Planck equation that describes the evolution of the probability distribution of Q-values in the agent population is derived. To the best of our knowledge, this is the first time to show the Q-learning dynamics under an n-agent setting can be described by a system of only three equations.